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Abstract 

It is known that there is a dichotomy in the performance of model selectors. 
Those that are consistent (having the "oracle property") do not achieve the 
asymptotic minimax rate for prediction error. We look at this phenomenon 
closely, and argue that the set of parameters on which this dichotomy occurs 
is extreme, even pathological, and should not be considered when evaluating 
model selectors. We characterize this set, and show that, when such parameters 
are dismissed from consideration, consistency and asymptotic minimaxity can 
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be attained simultaneously. 

Keywords: AIC; BIC; Consistency; Contiguity; Local alternative; Minimax-rate op- 
timality. 
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1 Introduction 



Model selection is an important area of statistical practice and research. However, 
model selection often represents a first step towards our main goal, which may be 
estimation or prediction. A stylized scheme is the following: first use a model-selection 
procedure to select a model, and then proceed with inference conditionally on the 
chosen model. This method leads to so-called "post-model-selection" estimators (or 



predictors). Our conc ern here is with t 



The review paper 



l e asym ptotic risk of such procedures. 



Leeb fc Potscher 



(I2OO5I ) argues that the (data-driven) model 



selection ste p typically has dramatic effect on the sampling properties of the estima- 



tors; see also 



Leeb fc Potscherl (120061 ). These properties are quite different from their 



single-model counterpart, and cannot be ignored even when the sample size is large 
and when the model selector is consistent. 

Although interest in the performance of post-model selection estimators has gained 
momentum over the last years, the problem has been around for a few decades and can 
be traced backed, in its essence, to Hodges' e stimator. For an interesting discussion 



of Hodges' estimator see 



van der Vaart 



(11998 



, Example 8.1) 



Within the context of regression functions, and for squared-error loss, 



YangI (I2OO5I) 



has shown that consistent model selection procedures, such as BIG, produce estima- 
tors which cannot attain the asymptotic minimax rate. Failure to attain this optimal 
rate extends also to model combination, or Bayesian model averaging with subjec- 
tively specified priors. On the othe r hand, it is known that AlC, which is inconsistent. 

Proposition 1). This tension between 



does attain the minimax rate; see 



lYand (120051 
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model consistency and inference-opt imality is sometimes referred to as the "AIC-BIC 
dilemma" . 

Attempts at overcoming this dilemma include adaptive model selection, which, 



unlike AIC or BIC, employs a data- driven pena 
optimality; for a brief account see lYangj (12005 



ty to achieve both consistency and 
, sec. 1.4). Recently, van Erven, 



Griinwald and Rooij introduced the notion of "switch distribution" , as an alternative 
to standard model selection methods, such as the Bayes factor and leave-one-out 
cross-validation, in an effort to combine the strengths of AIC and BIC; see their 2008 
Technical Report entitled Catching Up Faster by Switching Sooner: a Prequential 
Solution to the AIC-BIC Dilemma flarXiv:0807.1005i ^l [math. ST]). 

The remainder of the paper is organized as follows. In Section [2] we review the 
criteria for asymptotic comparison of tests based on power against local alternatives, 
and show how it relates to the prediction problem in linear regression. In Section 
|3] we revisit the result of Yang (2005) for the simple linear regression example. In 
particular we provide an evaluation of the proof he gave for his Theorem 1, and we 
link his sequence of alternatives to the categorization given in Lemma 12. 1[ This puts 
into perspective, and actually explains why the failure to attain the minimax rate 
occurs; we also show that the minimax rate is achieved for sequences of type 3 in 



Lemma 12.11 which are recognized as the only reasonable sequences for the asymptotic 
comparison of tests. In Section H] we comment on the use of contiguity for proving 
lack of minimax rate for consistent model selectors. Finally, Section [5] offers some 
concluding remarks. 
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2 Asymptotic Comparison of Tests 

In this section we review some results on asymptotic test comparison. In particular we 
are interested in the categorization of local alternatives, and how their convergence to 
the null value interacts with the power of a test. We then look at the simple example 
of model selection in linear regression. 

2.1 Categorizing Alternatives 

Consider a sequence of statistical models {Pn.e, G 0} for observations y„ : = 
(Ui, . . . ,yn), n = 1,2,..., where we want to test Hq : 6 = OqVsHi : 6 > 6q. If 
nn{9) is the power function of a test, for most reasonable tests it holds true that 
lim^^oo 7r„(^) = if 9 = 9o, and lim^^oo ''^ni9) = 1 ii 9 > 9^. This is to be expected 
because, with arbitrarily many observations, it should be possible to tell the null 
and the alternative apart with complete accuracy. This fact means that to compare 
tests asymptotically we should make the problem "harder" . One way to do this is to 
consider a sequence of testing problems 

Ho:9 = 9oVsH^^:9 = 9r„ (1) 

where 9n > 9q and 6'„ 9^ in specific ways. 

The Li-distance between Pn,eo ^-nd Pn,en provides a useful characterization of the 
power function associated with ([T]). Denote the Li-distance between two probabil- 
ity measures P and Q (having density p, respectively q, with respect to a common 
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measure /i) by 1 1 P — Q 1 1 • Then 

\\P-Q\\= j b-gM/i = 2sup|P(A)-Q(A)| =^2||P-Q||ry, 

where the second equahty follows from the well-known relationship between Li dis- 
tance and total variation norm ||P — QHtv- 

The following lemma relates the Li-distance to the power function. 



Lemma 2.1 ( Ivan per VaartI . Il998l . lemma 14.30) 



The power function iin of any test satisfies 

7ln{9) - TlM < ^\\Pn,e ' i^Mo 1 1 • (2) 

For any 6 and Oq there exists a test whose power function attains equality. 
The implications of Lemma 12.11 are 

1. If ||Pn.e„ ~ -fn,6ioll ~^ 2, then the sequence On is converging to 9q at a slow rate, 
so that the two hypotheses are strongly separated. In this case the difference 
^n(^) 7r„(6'o) tends to 1, which means that we can get all sort of tests; in 
particular, since equality can be attained, there exist a sequence of tests with 
power tending to 1 and size tending to (a perfect sequence of tests). 

2. If ||Pn,0„ — -Pn,0oll 0, then the sequence ^„ is converging to 6*0 at a fast rate, 
so that the two hypotheses are weakly separated. In this case the power of any 
sequence of tests is asymptotically less than the level (every sequence of tests 
is worthless). 
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3. If ||P„ 



n,6»o I 



is bounded away from and 2, then the sequence 9n is con- 



verging to ^0 ^ r^-te such that the two hypotheses are well separated. In this 
case, there exists no perfect sequence of tests, but not every test is worthless 
either. 



The consensus in t 
13.1) or van der Vaart 



l e liter ature, see for example iLehmann Sz Romano! (l2005l . sec 



(119981 . sec. 14.5), is that situation 3 is the only reasonable one 



for the comparison of tests, otherwise the problem is "asymptotically degenerate". 
For iid observations from smooth models, case 3 occurs when 0„ converges to at 
rate l/y/n. 

Easier calculation often results when using Hellinger distance rather than the Li- 

1 /2 

distance. The Hellinger distance between P and Q is H{P, Q) = {f {y/p — y/<l)^dfi^ , 
and we can rewrite its square as H^{P, Q) = 2 — 2A{P, Q), whe re A(P, Q) = J ^/yqdu 



is called the Hellinger affinity. The following inequality holds (Ivan der Vaart 



19981 ): 



H\P,Q) <\\P-Q\ \ <mm{2-A'{P,Q),2H{P,Q)}. 



(3) 



2.2 Example: Simple Linear Regression 

Here we look in detail at the simple testing problem considered by Yang (2005). For 
2 = 1, . . . , n let 



Ho : yi = ei and Hi : yi = (3xi + e^; (3 > 0, 



(4) 



where ~ iV(0,l). 

Recall that two sequences a„ and 6„ are said to be of the same order, written 
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&„,, when there exist constants < r < R < oo and an integer no such that, for n > Uq, 
r < |a„,/6„| < R. We start with a simple lemma whose proof is straightforward. 

Lemma 2.2 Let Pnfl he the probability measure associated with Hq and Pn.p that as- 
sociated with Hi in Assume that lim^^oo S"=i xj/n is a strictly positive constant, 
so that XliLi ^ n. The Hellinger affinity is 

^(Pn,0,Pn„/3)=exp|-^|jx,2|. (5) 

As a direct consequence of this lemma, we can characterize sequences /3„ as follows. 

1. if Pn = Cn/ ^yn, with c„ oo, then ||-Pn,o — -Pn.ftJI 2 and the two models are 
strongly separated; 

2. if Pn = Cn/ Vn, with c„ O"*", then | |Pn,o — -Pn,/3„| I and the two models are 
weakly separated; 

3. if Pn ^ (Vv^)' then ||-P„,o ~ -f«,/3nll is bounded away from and 2, so that the 
two hypotheses are well separated. 



Following iLehmann fc Romano! (120051 . p. 498) we conclude that also for this 



regression problem the problem of testing P„^o versus Pn,/3„ is degenerate unless 
X l/y/n. These are therefore the only meaningful local alternative sequences 
for evaluating tests. 

2.3 Consistency and Prediction 

The simple testing problem described in (jlj) can be cast as a model selector by defining 
An = {{xi, yi),i = 1, ... n : Hi is selected }, and estimating /5 with the post-model- 



selection estimator where /(A„) is the indicator function of the set An- For 

simphcity we take (3 to be the least squares estimator under Hi. 

Given {xi,yi),i = l,...n, we run our model selection procedure and then we 
predict at values x*,i = 1, . . . , m, where the x* may be the same as the original Xi, 
or not. The average prediction error is 



m / 1 ™ \ 

m ^-^ \ m ^-^ j 

i=l \ i=l / 



which shows why it doesn't matter whether we predict x* or Xj, or how many x* we 
predict (as long as m/n is finite as n — >• oo). Taking expectations gives the predictive 
risk function 

Rmi^^An) = (^f2{x:Y^Ep{P-pi{An)Y 
1 ™ \ 

i=l ) 

The predictor attains the asymptotic minimax rate if n sup^ A„) — > constant 
as n — s> oo. The quantity nRm{f3,An) is called the scaled risk (function). Recalling 
that Yl^ii^i)"^ ^ ""^5 only need to be concerned with 

nsnpRUP,An)-n[Efs{{(3-^yi{An)} + ^^Pfs{A'J]. (6) 

The first term is bounded by nE^{(/? — = na^/ ^o its limit is a positive 

constant. Thus, the model selector achieves the minimax rate if and only if 

sup n/9^P/3(/l^) — > constant as ra — oo. (7) 

P 

Yang (2005) found a sequence /3„ converging to zero, equivalently a sequence of alter- 
native models converging to the null-model, slowly enough to have — > oo, but at 
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the same time fast enough to "confuse" the model selector, leading it to choose the 
null model; this keeps P^^^(A^) away from zero, and actually arbitrarily close to one, 
so that the minimax rate is not achieved. 

3 The Predict ion/ Minimax Rate Conflict 

In this section we describe the sequence chosen by Yang, to establish his result. 
We also show that, with a minor modification, a similar sequence attains the minimax 
rate. We then look a bit closer at Yang's sequence, showing how it unfairly "confuses" 
the model selector. 

3.1 Yang's Sequence 

For the model selection problem described in (jlj), the model selector is consistent if 

P{An) ^ if i/o is true and P(A„) ^ 1 if iJi is true. 

Yang (2005) shows that a consistent model selector cannot achieve the minimax pre- 
diction rate using the following argument. He considers a sequence of UMP tests for 
the hypotheses (jl]) having power function 

^n{f3) = Pr^iY, x,y, > d„} = Pt,{Z > " 'l^""' }, (8) 

where Z ~ A^(0, 1), and equates Pr^=o(^ri) with 7r„(0), for each n. Note that the set 
A* = {{xi,yi) : J^i^iVi > dn} defines a model selector based on the UMP test, where 
we would estimate /3„ with the least squares estimator if this set occurred. 
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The requirement of consistency means that we must have vr„(0) which imphes 

oo. (9) 



Yang's objective is to find a sequence of alternatives — > 0+ such that 

CI: npl ^oo and C2 : (1 - 7rn(/3n)) ^ constant > 0, (10) 

which would imply that nP'^PT/s^^A^) — > oo (because the test is UMP), and hence 
lead to the conclusion that a consistent model selector does not attain the minimax 
rate, that is, it violates ([7]). Specifically, Yang's choice for /3„ is 

^ l2' ^^^^ 

which satisfies CI and C2. (There is a typo in Yang (2005), as noted by the author on 
his webpage: on lines 4 and 6 of page 947 the factor 2 should be in the denominator.) 
We now look at sequence (fTTl) a bit more closely. Write 

0n = — " , withc„ = " — » oo , (12) 

and, recalling that XliLi ^ immediately conclude that we are in scenario 1 of 
Lemma [2. 2 1 namely that the two hypotheses are strongly separated. In this case any 
type of test can occur, and with Yang's choice the power function is asymptotically 



zero: 



n Id 

TTnm = Pr^„{^a:.i/.>ci4 = Pr{Z>-^=^}^0, 



1=1 



which follows from and the fat that Xii/i ~ N(/3„ x^, x^) . 

Yang's sequence for Pn thus produces a worthless test, since both the size (by 
assumption) and the power (by construction) tend to zero. In this case we know that 
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there even exists a perfect sequence of tests (we will construct one below). Clearly 
Yang's argument holds for any sequence /3„ = 2 , 0<6<1. If 6=1 we get 

1/2, which would still support Yang's argument, although the test is no 
longer worthless (but actually rather poor because its power is fixed at 1/2 for each 
n). 

Looking at Yang's result from a testing perspective reveals one of its weak points. 
His result holds because he chooses a sequence of alternatives that, while producing 
asymptotically a strong separation between the two models, converges to the null 
along a path leading to a worthless, or at best a mediocre, test. This happens despite 
the test being UMP, and despite the fact that, by a minor modification, one could 
get a perfect sequence of tests, as the following example shows. 

Example 3.1 Consider the UMP test for the hypotheses Choose 

P, = (l + b')=^,b'>0. (13) 

1^1=1 

Then 

TT^m = Ft{Z > -b' J^ } - 1, 

which follows from ^ and the fact that b' > 0. As a consequence, by choosing a 
sequence of alternatives which is structurally equivalent to Yang's, although uniformly 
larger by a factor (1 + b')/b, b' > 0, < b < 1, we get a perfect sequence of tests. 

□ 

With the choice of sequence (fT3|) . 1 — vr„(/3n) goes to zero exponentially fast, and 
it is easy to show that n/3,^(l — 7r„(/5„)) 0, instead of going to 00 as under Yang's 
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choice (fTTI) . Thus, under this sequence of alternatives, we even beat the minimax 
rate! Example 13.11 reinforces the view that Yang's result is based on a rather artificial 
sequence. Next we provide further insight into his choice of sequence 



3.2 Confusing the Model Selector 

In Section [2l3] we noted that Yang's sequence /5„ was constructed in such a way as to 
"confuse" the model selector. We now look a bit more closely at this claim. 

What happens with Yang's sequence is that, as n — >^ oo, all of the mass of the 
distribution of Yli ^iUi is concentrated in the acceptance region of the test (that is, 
in v4^), even when (3n > 0, making the "correct" decision that of accepting Hq. To 
see this, recall that, for (3 = the UMP test-statistic J^i^iVi distributed as 
x^, J2i ^D- Consider the probability 



which grows arbitrarily close to 1 as M increases. Now set /?„ = b ^n" -2 , < b < 1 
as in Yang's choice. Then, for any M > 0, /3„ XliLi -^^ + ^ \/^a=i^ < eventually 
(that is, as n grows), because the previous inequality is equivalent to 



which holds true since the right-hand-side tends to infinity because of ([9]). But this 
means that the support of the UMP-test statistic under this sequence of alternatives 
is eventually disjoint from the iJi- acceptance region of the test: this is why the test 
is fooled and chooses Hq incorrectly, with probability tending to one. Notice that if 
b = 1 then does not hold. 




M <{l-b) 



(14) 
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Finally, for the sequence with /3„ = (1 + h') ^ , b' > 0, then PnYl^=i^l ~ 
Li ^1 > dm eventually, because the previous inequality is equivalent to 

_M>-b'^ ^ , (15) 

which holds true since the right-hand-side tends to —oo because of (I9l). This means 
that the support of the UMP-test statistic under this sequence of alternatives is 
eventually contained in the -ffi-acceptance region of the test: the test correctly chooses 
Hi with probability tending to one, and thus chooses Hq with probability tending to 
zero. The model-selector estimator attains the minimax rate. 



4 Contiguous Sequences 



Leeb fc Potscherl (|2005| . Appen dix C, 



parable to that of Theorem 1 in 



ropo sition C.l) present a result which is com- 



Yangl (120051 ). They consider a linear regression model 



with mean structure axii + I3x2i under the unrestricted case, and mean structure axu 
under the reduced model. They deal, among other things, with the scaled risk, under 
squar e d-erro r loss, of the post-model selection estimator (least squares) of a. As in 



Yang] (120051 ). they claim that its supremum diverges to infinity whenever the model 



selection procedure is consistent. Although Yang is concerned with prediction and not 
estimation, the connection between the two results is apparent in the case of normal 
errors. Yet, Leeb and Potscher's argument is quite different from Yang's, because it 
relies on the n o tion o f contiguity. F or an introduction to t he no tion of contiguity, see 



van der Vaart 



( 19981 . sec. 6.2) and 



Lehmann fc Romano! (120051 . sec. 12.3). Here we 



revisit Yang's problem using the notion of contiguity. We provide an evaluation of 
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this technique for the problem at hand and raise some critical issues. 
Let Pn and Q„ be measures on a measurable spaces 

Definition 4.1 The sequence Qn is contiguous with respect to the sequence Pn if 
Pn{An) — > implies Qni^n) for every sequence of measurable sets An- This is 
denoted Qn < Pn- 

One can regard contiguity as the asymptotic analogue of the classic notion of absolute 
continuity of measures. The strength of contiguity stems from the fact that Qn-hmit 
law of random vectors Un : fin ^ can be obtained from suitable P„-limit laws; 
the usefulness of such result is apparent when the latter calculations are much easier 
than the former. If Qn is contiguous with respect to Pn, and viceversa, then we write 

Qn < >Pn- 

The following result considers the model selection problem discussed by Yang, and 
relates it to the notion of contiguity. 

Proposition 4.1 Consider the problem described in 0). Let Pn^ be the sequence of 
probability measures under the null model Hq, and Pn,i3n be the sequence of probability 
measures corresponding to the local alternative models Hin : yi = PnXi + e^, /3„ > 0, 
/3n — 0"''. Then Pn,i3n <1 -Pn.o if CLnd only if /3„ = 0{1/ y/n); additionally Pn,^„ < >Pn,o 
under the same condition. 

Proof. The proof is essentially the same as that for proving contiguity of the joint 
distribution of n iid observations from a Normal with mean C,n and variance 1 with 
respect to the joint distribution of n iid with observations from a Normal with mean 
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and variance 1; see lLehmann fc Romano! (120051 . examples 12.3.3 and 12.3.6). To see 



why, simply notice that the likelihood ratio is 

n n 

dPn,f3„/dPn,o = exp{/3n^Xiyi - (/3^/2) ^x-}. 

i=l 1=1 

Under P„,o, DILi ^iVi ~ ^(0> Yh=i ^1), and thus, 

n n n n 

j=l j=l i=l i=l 

again under P„,o- So the only difference between the simple regression case we are 
discussing and the iid case from a Normal is that the former has ^1^=1 ^1 while the 
latter has n. Since these two quantities are asymptotically of the same order, one can 
use the same argument in either case. ■ 

From Proposition 14.11 it appears that -Pn,/3„ < Pn,o if and only if the sequence n/?^ 
remains bounded; under the same condition mutual contiguity holds. In particular 
even if — > 0, but at a slower rate than 1/y/n, as in Yang's case, see (fT2|) . then 
contiguity of -P„,,/3„ fails. Notice that contiguity holds also if Pn = o{l/^/n), because 
Pn Sr=i ^"i and hence is bounded. However this case is of no interest 

for proving failure to attain the minimax rate, because condition CI in (fTOj) is not 
satisfied (that is, np^ does not diverge but actually goes to zero). 

How can we use contiguity to obtain a result similar to Yang's? Here is the idea. 
Yang's result obtains if we show that 

lim nplFip {A'^J = oo 

To exploit contiguity, /3„ must be of order Xj^fn. Set for definiteness /9„ = t/ ^/n, for 
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some positive fixed r. We get 

lim Prp_ {A^} = lim Vvp^^,{An} = 1, 

where the first equahty sign follows from contiguity of Pn,r/^ with respect to Pnfi-, 
while the second is a consequence of the assumed consistency of the model selector 
(recall that means accepting Hq : [3 = . Therefore 

lim nr\l/^fVvp^ {A'^^} = r\ (16) 

At this stage it would seem that the minimax rate is attained under this sequence, 
because the limit, however large, is finite. To circumvent ( |T6l) . and get the opposite 
conclusion that t he rat e is actually infinite, one ought to apply the argument in 
Leeb fc Potscherl (120051 . p. 59), and let r grow arbitrarily large (technically this 
amounts to take a further limit r oo). However, there is a subtle difference 
between this argument and Yang's result. 

First of all, to conclude that lim^^oo ''^/^^Prp^, ^„ {^n} = ^ one should prove that, 
for any R> 0, there exists an uq such that 

n>no^nPlFrp^^^^{A^J>R. (17) 

Having set /9„ = r/y/n, condition (fT7|) translates to 

n > no ^ Prp,^ > R/r'. (18) 

Since r is fixed, condition 0181) can be easily violated choosing for instance R> r^. 

Secondly, and possibly more importantly, the argument based on contiguity con- 
veys the misconception that sequences of alternatives Pn of order 1 / y/n can fail to 

17 



attain the minimax rate. This would be quite surprising because, on the contrary, it 
is well known that this type of sequences is the only one which makes sense for asymp 



totic comparison of tests, as lucidly remarked, for instance, in 
(I2OO5I, example 12.3.6). 



Lehmann fc Romano 



In the light of the above remarks, and of (fT6|) . it should be clear that sequences 
of alternatives of order 1 / i/n do achieve the minimax rate. 

5 Concluding Remarks 

In this paper we have cast the so-called AIC-BIC dilemma into perspective. On 
the one hand it is true that estimators and predictors based on consistent model 
selection procedures m ay lea d to a n infinite scaled risk, thus failing to attain the 



usual minimax rate, as lYangj (|2005| ) showed. On the other hand, this phenomenon 
occurs only for sequences of alternatives which are strongly separated from the null 
model (this inflates the bias when the null model is chosen, while the alternative 
holds). But such sequences are well known to be of no use in asymptotic comparison 
of testing procedures, because they always admit a perfect sequence of tests (the 
power goes to 1 while the size goes to zero). 

Additionally, the non-attainment of the minimax rate takes place only for a spe- 
cific subset of these sequences, namely those whose support under the alternative 
is eventually fully contained in the null-acceptance region. This explains why the 
problem occurs: the selector (not surprisingly!) chooses the null model, although the 
alternative holds. Finally we have argued that contiguity arguments have little to 



say with regard to the AIC-BIC dilemma: contiguity is synonymous with sequences 
of alternatives converging to the null at the appropriate rate 1 / ^Jn: no pathological 
behaviour can occur in this case. 
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